from sklearn_benchmarks.reporting.hp_match import HpMatchReporting

scikit-learn vs. ONNX Runtime

reporting = HpMatchReporting(other_library="onnx", config="config.yml", log_scale=True)
reporting.make_report()

We assume here there is a perfect match between the hyperparameters of both librairies. For a given set of parameters and a given dataset, we compute the speedup time scikit-learn / time onnx. For instance, a speedup of 2 means that onnx is twice as fast as scikit-learn for a given set of parameters and a given dataset.

KNeighborsClassifier_brute_force

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=brute.

Mismatches between validation scores

WARNING! Mismatch between validation scores for 12 predictions (75.0%).
The observed differences can be found in the diff_accuracy_scores column.
The chosen difference threshold is 0.001 for accuracy_score.
See details in the dataframe below.

	estimator	function	diff_accuracy_scores	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	algorithm	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup
1	KNeighborsClassifier_brute_force	predict	0.337	100000	1000	100	1.994884	0.172649	NaN	0.000401	0.001995	brute	-1	1	0.663	0.354387	0.005473	1.000	5.629111	5.629782
2	KNeighborsClassifier_brute_force	predict	0.243	100000	1	100	0.022639	0.002272	NaN	0.000035	0.022639	brute	-1	1	1.000	18.982136	0.359374	0.757	0.001193	0.001193
4	KNeighborsClassifier_brute_force	predict	0.125	100000	1000	100	2.951273	0.047169	NaN	0.000271	0.002951	brute	-1	5	0.757	19.146237	0.042717	0.882	0.154144	0.154144
7	KNeighborsClassifier_brute_force	predict	0.118	100000	1000	100	2.199261	0.025263	NaN	0.000364	0.002199	brute	1	100	0.882	0.354428	0.013963	1.000	6.205100	6.209914
8	KNeighborsClassifier_brute_force	predict	0.243	100000	1	100	0.020690	0.001177	NaN	0.000039	0.020690	brute	1	100	1.000	19.135550	0.244916	0.757	0.001081	0.001081
10	KNeighborsClassifier_brute_force	predict	0.219	100000	1000	100	2.924734	0.059220	NaN	0.000274	0.002925	brute	-1	100	0.882	19.682121	0.255039	0.663	0.148599	0.148611
13	KNeighborsClassifier_brute_force	predict	0.243	100000	1000	100	2.188334	0.040193	NaN	0.000366	0.002188	brute	1	5	0.757	0.272164	0.006575	1.000	8.040485	8.042831
14	KNeighborsClassifier_brute_force	predict	0.078	100000	1	100	0.020871	0.001000	NaN	0.000038	0.020871	brute	1	5	1.000	4.029837	0.085303	0.922	0.005179	0.005180
16	KNeighborsClassifier_brute_force	predict	0.266	100000	1000	100	1.223682	0.020244	NaN	0.000654	0.001224	brute	1	1	0.663	4.128764	0.072914	0.929	0.296380	0.296426
19	KNeighborsClassifier_brute_force	predict	0.104	100000	1000	2	1.747191	0.032236	NaN	0.000009	0.001747	brute	-1	1	0.896	0.268770	0.006979	1.000	6.500682	6.502873
20	KNeighborsClassifier_brute_force	predict	0.078	100000	1	2	0.005002	0.002656	NaN	0.000003	0.005002	brute	-1	1	1.000	4.012612	0.083597	0.922	0.001247	0.001247
22	KNeighborsClassifier_brute_force	predict	0.026	100000	1000	2	2.764815	0.041363	NaN	0.000006	0.002765	brute	-1	5	0.922	3.994434	0.046559	0.896	0.692167	0.692214

Raw results

predict

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.995	0.173	0.000	0.002	-1	1	0.663	0.354	0.005	1.000	5.629	5.630	See	See
2	KNeighborsClassifier_brute_force	predict	100000	1	100	0.023	0.002	0.000	0.023	-1	1	1.000	18.982	0.359	0.757	0.001	0.001	See	See
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.951	0.047	0.000	0.003	-1	5	0.757	19.146	0.043	0.882	0.154	0.154	See	See
5	KNeighborsClassifier_brute_force	predict	100000	1	100	0.025	0.003	0.000	0.025	-1	5	1.000	0.353	0.013	1.000	0.072	0.072	See	See
7	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.199	0.025	0.000	0.002	1	100	0.882	0.354	0.014	1.000	6.205	6.210	See	See
8	KNeighborsClassifier_brute_force	predict	100000	1	100	0.021	0.001	0.000	0.021	1	100	1.000	19.136	0.245	0.757	0.001	0.001	See	See
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.925	0.059	0.000	0.003	-1	100	0.882	19.682	0.255	0.663	0.149	0.149	See	See
11	KNeighborsClassifier_brute_force	predict	100000	1	100	0.028	0.005	0.000	0.028	-1	100	1.000	0.363	0.013	1.000	0.076	0.076	See	See
13	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.188	0.040	0.000	0.002	1	5	0.757	0.272	0.007	1.000	8.040	8.043	See	See
14	KNeighborsClassifier_brute_force	predict	100000	1	100	0.021	0.001	0.000	0.021	1	5	1.000	4.030	0.085	0.922	0.005	0.005	See	See
16	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.224	0.020	0.001	0.001	1	1	0.663	4.129	0.073	0.929	0.296	0.296	See	See
17	KNeighborsClassifier_brute_force	predict	100000	1	100	0.021	0.002	0.000	0.021	1	1	1.000	0.274	0.007	1.000	0.075	0.075	See	See
19	KNeighborsClassifier_brute_force	predict	100000	1000	2	1.747	0.032	0.000	0.002	-1	1	0.896	0.269	0.007	1.000	6.501	6.503	See	See
20	KNeighborsClassifier_brute_force	predict	100000	1	2	0.005	0.003	0.000	0.005	-1	1	1.000	4.013	0.084	0.922	0.001	0.001	See	See
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	2.765	0.041	0.000	0.003	-1	5	0.922	3.994	0.047	0.896	0.692	0.692	See	See
23	KNeighborsClassifier_brute_force	predict	100000	1	2	0.007	0.005	0.000	0.007	-1	5	1.000	0.283	0.007	1.000	0.026	0.026	See	See

KNeighborsClassifier_kd_tree

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=kd_tree.

Mismatches between validation scores

WARNING! Mismatch between validation scores for 12 predictions (75.0%).
The observed differences can be found in the diff_accuracy_scores column.
The chosen difference threshold is 0.001 for accuracy_score.
See details in the dataframe below.

	estimator	function	diff_accuracy_scores	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	algorithm	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup
1	KNeighborsClassifier_kd_tree	predict	0.071	1000000	1000	10	0.891720	1.171892	NaN	0.000090	0.000892	kd_tree	-1	1	0.929	2.997673	0.262560	1.000	0.297471	0.298609
2	KNeighborsClassifier_kd_tree	predict	0.054	1000000	1	10	0.002790	0.000454	NaN	0.000029	0.002790	kd_tree	-1	1	1.000	127.169801	0.000000	0.946	0.000022	0.000022
4	KNeighborsClassifier_kd_tree	predict	0.005	1000000	1000	10	1.141847	0.397180	NaN	0.000070	0.001142	kd_tree	-1	5	0.946	128.199489	0.000000	0.951	0.008907	0.008907
7	KNeighborsClassifier_kd_tree	predict	0.049	1000000	1000	10	5.498338	0.533157	NaN	0.000015	0.005498	kd_tree	1	100	0.951	3.135518	0.319834	1.000	1.753566	1.762665
8	KNeighborsClassifier_kd_tree	predict	0.054	1000000	1	10	0.003219	0.000686	NaN	0.000025	0.003219	kd_tree	1	100	1.000	127.138928	0.000000	0.946	0.000025	0.000025
10	KNeighborsClassifier_kd_tree	predict	0.022	1000000	1000	10	3.021899	0.291408	NaN	0.000026	0.003022	kd_tree	-1	100	0.951	126.774528	0.000000	0.929	0.023837	0.023837
13	KNeighborsClassifier_kd_tree	predict	0.054	1000000	1000	10	1.672402	0.245720	NaN	0.000048	0.001672	kd_tree	1	5	0.946	0.005896	0.000206	1.000	283.629602	283.801915
14	KNeighborsClassifier_kd_tree	predict	0.089	1000000	1	10	0.001587	0.000485	NaN	0.000050	0.001587	kd_tree	1	5	1.000	0.044325	0.003629	0.911	0.035812	0.035932
16	KNeighborsClassifier_kd_tree	predict	0.035	1000000	1000	10	0.904323	0.193756	NaN	0.000088	0.000904	kd_tree	1	1	0.929	0.067247	0.002510	0.894	13.447729	13.457091
19	KNeighborsClassifier_kd_tree	predict	0.109	1000	1000	2	0.033206	0.020860	NaN	0.000482	0.000033	kd_tree	-1	1	0.891	0.005599	0.000693	1.000	5.931034	5.976345
20	KNeighborsClassifier_kd_tree	predict	0.089	1000	1	2	0.003056	0.000374	NaN	0.000005	0.003056	kd_tree	-1	1	1.000	0.042173	0.002047	0.911	0.072468	0.072554
22	KNeighborsClassifier_kd_tree	predict	0.020	1000	1000	2	0.024285	0.001525	NaN	0.000659	0.000024	kd_tree	-1	5	0.911	0.041578	0.002049	0.891	0.584086	0.584795

Raw results

predict

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.892	1.172	0.000	0.001	-1	1	0.929	2.998	0.263	1.000	0.297	0.299	See	See
2	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.000	0.000	0.003	-1	1	1.000	127.170	0.000	0.946	0.000	0.000	See	See
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.142	0.397	0.000	0.001	-1	5	0.946	128.199	0.000	0.951	0.009	0.009	See	See
5	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	0.000	0.003	-1	5	1.000	2.960	0.217	1.000	0.001	0.001	See	See
7	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	5.498	0.533	0.000	0.005	1	100	0.951	3.136	0.320	1.000	1.754	1.763	See	See
8	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	0.000	0.003	1	100	1.000	127.139	0.000	0.946	0.000	0.000	See	See
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	3.022	0.291	0.000	0.003	-1	100	0.951	126.775	0.000	0.929	0.024	0.024	See	See
11	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.006	0.002	0.000	0.006	-1	100	1.000	2.964	0.186	1.000	0.002	0.002	See	See
13	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.672	0.246	0.000	0.002	1	5	0.946	0.006	0.000	1.000	283.630	283.802	See	See
14	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.002	0.000	0.000	0.002	1	5	1.000	0.044	0.004	0.911	0.036	0.036	See	See
16	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.904	0.194	0.000	0.001	1	1	0.929	0.067	0.003	0.894	13.448	13.457	See	See
17	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.001	0.000	0.000	0.001	1	1	1.000	0.006	0.000	1.000	0.188	0.188	See	See
19	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.033	0.021	0.000	0.000	-1	1	0.891	0.006	0.001	1.000	5.931	5.976	See	See
20	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.003	0.000	0.000	0.003	-1	1	1.000	0.042	0.002	0.911	0.072	0.073	See	See
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.024	0.002	0.001	0.000	-1	5	0.911	0.042	0.002	0.891	0.584	0.585	See	See
23	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.002	0.000	0.000	0.002	-1	5	1.000	0.006	0.000	1.000	0.377	0.378	See	See

HistGradientBoostingClassifier_best

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: learning_rate=0.01, n_iter_no_change=10.0, max_leaf_nodes=100.0, max_bins=255.0, min_samples_leaf=100.0, max_iter=300.0.

Mismatches between validation scores

WARNING! Mismatch between validation scores for 1 prediction (100.0%).
The observed differences can be found in the diff_accuracy_scores column.
The chosen difference threshold is 0.001 for accuracy_score.
See details in the dataframe below.

	estimator	function	diff_accuracy_scores	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	...	max_iter	max_leaf_nodes	min_samples_leaf	n_iter_no_change	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup
1	HistGradientBoostingClassifier_best	predict	0.176	100000	1000	100	0.127203	0.005715	300	0.006289	...	300	100	100	10	0.824	0.471227	0.018186	1.0	0.269939	0.27014

1 rows × 23 columns

Raw results

predict

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	HistGradientBoostingClassifier_best	predict	100000	1000	100	0.127	0.006	300	0.006	0.0	0.824	0.471	0.018	1.0	0.27	0.27	See	See